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CAMERA-BASED PRECRASH DETECTION SYSTEM 
BACKGROUND OF THE INVENTION 

[0001] The present invention is directed to a method for detecting road users and obstacles 
oh the basis of camera images, and to a device for detecting road users and obstacles on the 
basis of camera images. 

[0002] By detecting an imminent collision between a road user and the vehicle of an 
observer at an early stage, one can improve the safety of vehicle occupants as well as of the 
potential other party involved in the collision. The time savings that can be gained by visually 
detecting and evaluating the observable region in front of the observer's vehicle permits 
stepped reactions of the vehicle occupant safety systems (e.g., a gentle firing of the airbag), or 
makes reactions possible in the first place to protect the other party in a collision (such as 
raising the engine hood in the case of collision involving a pedestrian). Developments 
culminating in switchable crash structures have made it more important than ever to know the 
type of other party involved in the accident (truck, passenger car, motor cycle, pedestrian). 

[0003] Current developments in precrash sensors are directed, inter alia, to the analysis of 
methods based on infrared lasers, ultrasound, or radar sensors. The drawbacks of these 
systems are due, in part, to their limited range (ultrasound, infrared laser) and to their inability 
to identify the other party potentially involved in a collision, along the lines of a reliable type 
classification (truck, passenger car, motor cycle, person). In radar-based systems, inter alia, 
non-metallic objects (e.g.: people or trees) are not able to be reliably detected by inexpensive 
sensors suited for use in vehicle applications. However, adapting an optimally stepped reaction 
of safety systems to imminent collisions requires reliable detection and dependable typing. 
Within the framework of such a stepped reaction, for example in the event of a collision with a 
pedestrian, active measures should be taken to ensure the pedestrian's safety. In this 
connection, one can conceive of rapid changes in the vehicle body form to minimize the 
probability of serious head or leg injuries. However, the basic condition for activating these 
measures is that the safety system be able to reliably detect road users and classify them 
according to type (e.g.: passenger car, truck, bicycle riders, pedestrians). 




[0004] Generally, methods for interpreting image scenes endeavor to not only obtain purely 
two-dimensional image information, but multi-dimensional scene information as well. They do 
so already in the first step, using a mostly costly sensor (stereo sensor or high-resolution radar 
or lidar). In order to detect objects, these methods are based, however, on models, above all 
with respect to the position and orientation of potential targets, as well as with respect to a 
predefined, fixed geometry of the orientation to the sensor and surroundings. In practice, it is 
often ascertained, however, that these models and assumptions frequently do not conform with 
actual conditions, so that misinterpretations result. 

SUMMARY OF THE INVENTION 

[0005] An object of the present invention is to devise a method and a suitable device which 
will make it possible to detect road users on the basis of camera images, to determine their 
distance from the observer, and to classify the road users. 

[0006] The present invention provides a method for detecting road users and obstacles on 
the basis of camera images, in order to determine their distance from the observer and to 
classify them, wherein a classifier designed for detecting road users and obstacles identifies 
regions within a two-dimensional camera image that is not resolved with respect to distance. 
In a subsequent step, these regions, so identified, are marked and then ranged using a distance- 
measuring sensor with respect to their distance from the observer. Selected regions are 
subsequently fed to a type classification to identify the road users or obstacles. In this case, 
the recording of data, organized into a plurality of steps, and object identification facilitate the 
use of conventional sensory technology and, at the same time, offer the potential of a real-time 
implementation. 

[0007] The present invention also provides a device for detecting road users and obstacles 
on the basis of camera images, to determine their distance from the observer, and to classify 
them, wherein a mono-image camera 1 is coupled to a distance-measuring sensor unit 3. In 
each case, this coupling has an interposed or downstream classifying unit 2 and 5, respectively. 

[0008] The method according to the present invention identifies regions within a camera 
image, in which road users or obstacles are situated. It advantageously suffices, in this context, 
that this camera image contains purely two-dimensional image information without any 
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distance resolution. The identification is carried out using a classifier specially trained for the 
road users and obstacles to be detected. In a subsequent step, the regions so identified are then 
marked and ranged using a distance-measuring sensor with respect to their distance from the 
observer. Selected regions are subsequently fed to a type classification to precisely determine 
the type of road user or obstacle. 

10009] A device suited for implementing this method includes a mono-image camera, which 
is coupled to a distance-measuring sensor unit. In each case, this coupling has an interposed or 
downstream classifying unit. 

[0010] In one advantageous embodiment of the device, the downstream classifying unit used 
for classifying types has an upstream selection unit connected in incoming circuit thereto, with 
whose assistance the number of regions to be classified can be controlled. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[001 1] Fig. 1 schematically depicts one advantageous refinement of a device for 
implementing the method of the present invention. 

DETAILED DESCRIPTION 

[0012] Figure 1 schematically depicts one advantageous refinement of such a device for 
implementing the method of the present invention. Here, mono-image camera 1 makes image 
data 10 available to a classifying unit 2, which identifies image regions containing the road 
users or obstacles and informs distance-measuring sensor unit 3 of the corresponding 
positional data. Sensor unit 3 then measures these regions with respect to their distance from 
the observer. In so doing, these measuring data 30, together with data 20 from first 
classifying unit 2, are available to a selection unit 4. Selection unit 4 can control the data flow 
to the downstream unit for classifying selected road users or obstacles 5 by type. Image data 
40 selected by the selection unit are transmitted to classifying unit 5 for classification. Results 
50 of this type classification are advantageously made available to a risk calculator connected 
to the classifying unit, so that the risk calculator can decide whether to initiate reactions that 
meet the requirements of the situation. 
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[0013] It is also conceivable, in another advantageous embodiment of the device according 
to the present invention, for selection unit 4 to be omitted and, basically, for all data 20 and 30 
to be fed directly to classifying unit 5. 

[0014] The method according to the present invention can be devised quite advantageously 
to identify road users and obstacles through the use of a hyperpermutation network within the 
framework of classifying unit 2. 

[0015] A network of this kind is able to localize regions belonging to a specific class (in this 
case road users and obstacles), at a high speed and on the basis of pixels, within image data 
10. The advantage of using simple, two-dimensional image information is not only apparent in 
the method's suitability for using a simple, inexpensive camera, but, in particular, also in the 
feasibility of using powerful classification algorithms under real-time conditions to analyze the 
entire image information. Since image information 10 supplied by a mono-image camera 1 is 
quite simple, it is possible, in contrast to conventional methods that mostly work on very 
complex data, to include every single image pixel in the classification. 

[0016] Within classifying unit 2, an algorithm, advantageously a box (boxcar-averaging) 
algorithm adapted to this task follows the actual classifier (for example the hyperpermutation 
network). This box algorithm combines and marks related regions of interest ROI, so that they 
can be fed for further processing. 

[0017] Within the framework of this further processing, the regions belonging to this ROI 
are marked and ranjged using a distance-measuring sensor 3, with respect to their distance 
from the observer. Radar systems or stereo camera systems are advantageously suited for this. 
Since the data acquired by these sensors are utilized purely for estimating distances and not for 
classifying types of road users or obstacles, there is no need to equip these sensors with 
extreme angular resolutions or with robust models requiring substantial computational outlay. 
Thus, one can preferably revert to using sensors which are already present in the vehicle and 
mainly geared towards other applications. 

[0018] It is also conceivable, however, in another advantageous embodiment of the method 
according to the present invention, for distances to be estimated in the area of the ROI using a 
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mono-image camera, in cooperation with a complex image analysis. Since, in this connection, 
only individual segments (ROI) of the entire image information need to be processed, large- 
volume computational work can be performed in real time using powerful processors. It 
would be especially beneficial, in this context, if this information could be directly obtained in 
a second processing step from image data 10 already supplied by mono-image camera 1. The 
system can also be implemented without the use of an additional distance-measuring sensor 3. 

[0019] In the method of the present invention, the information obtained by repeatedly 
measuring the distance of road users or obstacles from the user is advantageously used to 
determine the relative^ objects in relation to the observer. It is of particular 

benefit for distance-measuring sensor 3 to not only provide distance information, but velocity 
information as well (e.g.: double radar). This would enable the indirect velocity estimation to 
be omitted from the sequence of distance measurements. 

[0020] The distance and velocity information 30 is fed, together with image information 10, 
to a selection unit 4. Based on its default settings, this selection unit then decides which of the 
image data are to be supplied to a type classification within a downstream classifying unit 5. 
One can conceive of selection unit 4 being configured to basically supply all image data 
belonging to ROI to a type classification. However, one can also conceive of only those image 
data being transmitted which belong to the ROI having assigned road users or obstacles which 
meet specific criteria. In the process, the existing potential for danger is to be considered; thus, 
for example, the size of objects or the velocity at which they move towardJhejDbserYer, or 
also their relative velocity in general (for example, moving or stationary objects). 

[0021] For the actual type classification within classifying unit 5 which serves the purpose of 
precisely defining the type of road user or obstacle, one can fall back on classification 
algorithms specially trained for such objects. Advantageously suited for this are neural 
networks, such as a radial-basis function classifier or a support-vector machine. 

[0022] The method and device of the present invention are superbly suited for the early 
detection and subsequent assessment of accident situations (precrash detection). 
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